Photographic systems produce a wide range of image quality when operated by amateur, often referred to as “point-and-shoot”, photographers. If the photographic environment for a given scene is well suited to the image capture system (e.g. subjects are stationary and within the focus range, ambient light level is uniform and of sufficient intensity, and lens magnification is appropriate for the subject matter), good results are typically obtained. However, when these conditions are not present, image defects may be introduced due to failures in the capture or reproduction system, thereby reducing the quality of the final viewed image. To minimize the effects of suboptimal image capture conditions, camera designers have attempted to compensate by adding features intended to expand the range of light levels and distances where images can be captured. Unfortunately, these features often solve the primary problem, but add a secondary, sometimes severe, image defect.
For example, if the intensity of the ambient light is insufficient to provide adequate exposure, and the primary subject is located less than 20 feet from the camera, most built-in electronic flash units are able to provide auxiliary illumination sufficient to at least partially expose the primary subject. However, even if the primary subject now receives adequate illumination, the flash may introduce image defects.
As is well known in the art, the image defect known as redeye may occur when the angle between a narrow light source, the photographic subject, and the camera lens is less than approximately three degrees. This criterion is frequently met in flash exposures from compact cameras. The light from the flash enters the pupil nearly on-axis and propagates to the fundus of the eye, where it is reflected back out of the eye, having been colored red by the blood vessels in the fundus. The light exits the eye in a narrow cone, and if the camera lens falls within that cone, the red reflection will be recorded, and may appear in the final image as a red glow in the pupils, which is very undesirable in terms of image quality.
Redeye is more objectionable when the size of the pupil in the viewed image is larger and when the red saturation of the pupil is greater. The former may occur when the pupil is dilated, as occurs at low ambient light levels, or when the subject is rendered at a larger size in the image, for example due to shorter camera to subject distance, longer camera lens focal length, higher printing magnification (including zoom and crop), and/or shorter viewing distance. The primary techniques used in the camera to reduce or eliminate redeye are: increasing flash to lens separation; firing a preflash to transiently stop down the pupil in response to the bright light; and decreasing lens focal length and/or electronic zoom.
While all these methods are efficacious, all have associated disadvantages. Increased flash-lens separation may lead to more expensive and bulkier cameras and produces more noticeable shadows due to the farther off-axis lighting. After a preflash is fired, the eye requires half a second or more to respond fully, and during this delay between the preflash fire and the image capture, facial expressions of the subject often change in an undesirable fashion due to the annoyance and surprise of the preflash. The preflash also increases camera cost, reduces the power available during the main flash pulse, and increases battery consumption. Finally, restriction of optical or electronic zoom factors interferes with the photographer's ability to obtain the desired composition, with the subjects appearing large enough in the image to provide a pleasing rendition.
Given the disadvantages of the in-camera redeye reduction techniques summarized above, and the increased availability of digital printing devices capable of making corrections to selected portions of individual images, considerable effort has been directed towards the development of techniques for locating and correcting the redeye defect during the photofinishing (digital printing) process.
U.S. Pat. No. 5,748,764 issued 5 May 1998 teaches a method of locating and correcting the redeye image defect in an image. U.S. Pat. No. 5,892,837 issued 6 Apr. 1999, and related commonly assigned US. Pat. No. 6,292,574 issued Sep. 18, 2001 and U.S. Pat. No. 6,151,403 issued Nov. 21, 2000 generally describe additional methods suitable for locating human eyes in an image, and specifically describe locating and correcting the appearance of human eyes with the redeye image defect. These digital redeye removal techniques, while effective, are computationally intensive, and therefore increase the time required to optimally render and reproduce copies of captured images. The time required to perform these operations may in some cases be the rate limiting step in automated high speed printing operations. If the redeye defect location and correction processes are applied to every image in a customer order, even though only a portion of those images actually contain the defect, productivity and profit may be reduced. In addition, if computational time is spent searching for redeye defects in every image, other beneficial image processing operations such as tone scale mapping, digital noise reduction and sharpening may not be possible in the time interval allocated for each image. It is therefore desirable to be able to predict when redeye will occur, and to invoke the redeye location and correction processes only when needed.
From an extensive study to determine whether it is possible to predict from data collected at the time the original scene is photographed, the probability and severity of the redeye defect that will be present in the final image, it was discovered that the extent of the redeye defect depends primarily on the following factors: subject race, subject age, preflash illumination level, flash-to-lens separation, camera to subject distance, ambient light level, camera lens focal length, reproduction magnification, and final image viewing distance. In the present invention these factors are used to predict, on an image by image basis, the severity of the redeye defect, and that information is transferred from the camera to the photofinishing system where it commands the automatic printer control system, or in the case of human assisted printers, alerts the operator, to apply redeye defect location and correction techniques only when warranted, thereby improving picture quality and enhancing photofinishing productivity.
In addition to the redeye image defect, it is well-known that the physics of light intensity loss as a function of distance from a narrow source, such as an electronic flash tube, often leads to a defect in lighting contrast and consequently distorted tone reproduction in the final viewed image. Specifically, with every doubling of camera-to-subject distance, the light intensity per unit area on the subject drops by a factor of four. For example, if the primary subject is located 6 feet from the camera and the background is located 12 feet from the camera, the captured image of the background has an exposure level only one quarter that of the image of the primary subject. This causes the background to appear much darker than the primary subject does in the final viewed image. Because light falls off according to this squared function with respect to distance, the exposure difference between the primary subject and the background are often larger than illustrated above, particularly when images are captured outdoors at night or in large rooms. When presented with an image having a large exposure range (high contrast scene) with no knowledge of which portion of the scene is the primary subject, the exposure control system in the printer often calculates an average or area-weighted exposure that may excessively lighten the primary subject. This defect is particularly detrimental in pictures of people, whose faces are washed out and lack proper flesh reproduction.
If images of high contrast scenes universally contained overexposed primary subjects and underexposed backgrounds, as illustrated above, it would be practical to introduce a darken bias when printing all high contrast scenes. Unfortunately, there is a class of scenes known as backlight that are high in contrast, but have a subject-to-background exposure ratio that is opposite that of flash scenes. In the case of backlight scenes the illumination source is often behind the primary subject, or the primary subject is shaded by another object, such as a tree or a building, and therefore receives only a fraction of the ambient illumination. Consequently, the primary subject is underexposed relative to the background. In this case if the darkening bias needed to correct harsh flash scenes was applied, the already dark primary subject would be rendered even darker, having the effect of further reducing the image quality.
The information exchange (IX) feature of the Advanced Photo System offered by Eastman Kodak Company may make use of information collected at the time of image capture and passed to the printer to indicate whether, for the current image, the electronic flash was employed. (The Advanced Photo System specifications documents can be found at http://www.kodak.com/global/en/consumer/APS/redBook/specsIndex.shtml.) If the flash was fired, and a high contrast scene is inferred from the scanned image densities, a darkening bias can be applied to the image during printing. This information helps discriminate between backlight and harsh flash shots, and increases the probability that the primary subject will be printed to the proper lightness. However, because in both backlight and harsh flash scenes the dynamic range of the scene may exceed the tonal range of the print material, the primary subject and background can not be simultaneously rendered properly by invoking a full-image-field darken (in the case of harsh flash) or lighten (in the case of backlight) printing correction. This means that optical (analog) printing systems, which are only capable of producing full-field exposure corrections, can not produce optimal renditions of high contrast scenes.
Recent advances in digital image processing make practical methods for digitally segmenting the image field, analyzing the dynamic range, and adjusting tone reproduction (lightening or darkening) on an image area specific basis. By remapping the tone reproduction in this fashion, both the overexposed and underexposed portions of high contrast scenes can be rendered within the tonal range of the print material, thereby making the information in both regions visible in the final image. These digital area-specific tone scale remapping techniques, while effective, are computationally intensive, and therefore increase the time required to optimally render and reproduce copies of captured images. The time required to perform these operations may in some cases be the rate limiting step in automated high speed printing operations. If the tone scale remapping techniques are applied to every image in a customer order, even though only a portion of those images actually contain the defect, productivity and profit may be reduced. In addition, if computational time is spent searching for tone scale defects in every image, other beneficial image processing operations such as redeye location and correction, digital noise reduction and sharpening may not be possible in the time interval allocated for processing each image. It is therefore desirable to be able to predict when tone scale defects will be present, and to invoke tone scale remapping processes only when needed.
From a study to determine whether it is possible to predict from data collected at the time the original scene is photographed, the probability and severity of the tone scale defect that will be present in the final image, it was discovered that the extent of the tone scale defect depends primarily on the following factors: flash state (full,fill,off), primary subject light level, background light level, primary subject distance, background distance, and if available, state of the manual or automatic camera backlight exposure compensation control. In the present invention, these factors are used to predict, on an image by image basis, the severity of the tone scale defect, and that information is transferred from the camera to the photofinishing system where it commands the automatic printer control system, or in the case of human assisted printers, alerts the operator, to apply tone scale defect location and correction techniques only when warranted, thereby improving picture quality and enhancing photofinishing productivity.
If the ambient light level is not sufficient to provide adequate exposure, and the flash is deactivated or the primary subject is located beyond the maximum flash range, the image capture system will produce an underexposed image. In the case of film-based camera systems underexposure leads to latent image formation in primarily the most sensitive (fastest) layer, comprised of the largest silver halide grains. When processed and printed, images comprised of these fast, large grains permit a reproduction of the scene to be created, but the final viewed image typically contains noticeable grain structure, which masks fine detail and lowers the perceived image quality. The appearance of the grain, referred to more generally as image noise, becomes more objectionable when the reproduction magnification is increased, for example, in enlargements, pseudo-panoramic or pseudo-telephoto print formats.
In the case of digital still cameras (DSCs) with, for example, CCD or CMOS image sensors, the photographic sensitivity (exposure index) of the sensor may be adjusted automatically or manually, by the photographer, in response to the scene light level, to attempt to maintain adequate tone reproduction. The photographic sensitivity is adjusted by changing the gain of the sensor signal amplifier, taking into account the color temperature (white balance) of the ambient illuminant. When the ambient light level is high (bright scene), the amplifier gain is low, thereby producing a high (favorable) signal-to-noise ratio (SNR). When the ambient light level is low (dim scene), the amplifier gain is increased, which produces a low (unfavorable) SNR. When the gain is increased in this fashion, the tone reproduction of the image is improved relative to the standard amplifier gain; however, due to the low SNR, the final viewed image will typically contain noticeable noise, analogous to the grain in underexposed film images, which masks fine detail and lowers the perceived image quality. The appearance of noise defects becomes more objectionable when the reproduction magnification is increased, for example, in enlargements, pseudo-panoramic or pseudo-telephoto (electronic zoom) print formats.
Techniques such as those exemplified in The Sigma filter, described by Jong-Sen Lee in the journal article Digital Image Smoothing and the Sigma Filter, Computer Vision, Graphics, and Image Processing Vol 24, p. 255–269, 1983, are useful as noise reduction algorithms to enhance the visual appearance of the processed digital image. These digital area-specific noise reduction techniques, while effective, are computationally intensive, and therefore increase the time required to optimally render and reproduce copies of captured images. The time required to perform these operations may in some cases be the rate limiting step in automated high speed printing operations. If the digital noise reduction techniques are applied to every image in a customer order, even though only a portion of those images actually contain the defect, productivity and profit may be reduced. In addition, if computational time is spent searching for and correcting noise defects in every image, other beneficial image processing operations such as redeye location and correction, tone scale remapping and sharpening may not be possible in the time interval allocated for processing each image. It is therefore desirable to be able to predict when noise defects will be present, and to invoke digital noise reduction processes only when needed.
From an extensive study to determine whether it is possible to predict from data collected at the time the original scene is photographed, the probability and severity of the noise defect that will be present in the final image, it was discovered that the extent of the noise defect depends primarily on the following factors: Reproduction magnification; final image viewing distance; baseline exposure index film noise, or, in the case of DSCs, baseline exposure index sensor noise and the state of the manual or automatic DSC (R,G,B) exposure index control, which determines the sensor amplifier gain level; and the film or the sensor exposure level. In the present invention these factors are used to predict, on an image by image basis, the severity of the noise defect, and that information is transferred from the camera to the photofinishing system where it commands the automatic printer control system, or in the case of human assisted printers, alerts the operator, to apply noise defect location and correction techniques only when warranted, thereby improving picture quality and enhancing photofinishing productivity.
Even if the photographic environment provides ambient light that is uniform and of sufficient intensity to provide an exposure level that obviates the need for electronic flash or high-noise ambient captures, and the primary subject is within the focus range of the camera, the camera lens magnification provided by the normal lens (often defined as the diagonal dimension of the image capture frame) may be insufficient to capture an image of the primary subject that is the preferred size in the final viewed image. The size of the primary subject in the final viewed image is proportional to a quantity known as the angular magnification (AM) of the system, which can be characterized by the following equation:AM=[(Fl)(Mr)]/VdWhere:    Fl=camera lens focal length (specified in inches)    Mr=reproduction magnification (ratio of image to display size)    Vd=final image viewing distance (specified in inches)
The eye-to-display separation (viewing distance) has been found to vary with respect to final display size according to the following formula disclosed by the present inventors, in columns 43–44 of commonly-assigned U.S. Pat. No. 5,323,204:Vd=3.64+11.34[log10(D)]Where:    D=the diagonal dimension of the final display (specified in inches)
The most common method for increasing the AM involves the inclusion of telephoto or variable (zoom) focal length image capture optics on the camera. This approach produces larger subjects in the final viewed image by increasing the image capture magnification and maintaining a standard (full frame) printing magnification. Other methods involving pseudo-telephoto optical printing or electronic zoom digital printing are well known in the art. These techniques produce larger subjects in the final viewed image by increasing the printing magnification and cropping out a portion of the image frame, while retaining the standard print size (e.g. 4×6 inch) and the standard image capture lens focal length. Finally, by simply producing a larger (e.g. 8×10 inch) final image size, the reproduction magnification is increased, and therefore the AM and perceived subject size, even after including the longer final viewing distance, are also larger. The increase in AM provided by the aforementioned techniques may lead to a more pleasing composition, however, it also magnifies image blur resulting from inadequate lens depth-of-field, subject motion, and photographer hand tremor. The magnified image blur causes sharpness defects to be visible in the final viewed image.
Recent advances in digital image sharpness enhancement, as exemplified in U.S. Pat. No. 5,398,077, teach methods for digitally segmenting the image field, analyzing the content to separate signal and noise components, and boosting the sharpness on an image area specific basis. These digital area-specific sharpening techniques, while effective, are computationally intensive, and therefore increase the time required to optimally render and reproduce copies of captured images. The time required to perform these operations may in some cases be the rate limiting step in automated high speed printing operations. If the digital sharpening techniques are applied to every image in a customer order, even though only a portion of those images actually contain the defect, productivity and profit may be reduced. In addition, if computational time is spent searching for and correcting sharpness defects in every image, other beneficial image processing operations such as redeye location and correction, tone scale remapping and noise reduction may not be possible in the time interval allocated for processing each image. It is therefore desirable to be able to predict when sharpness defects will be present, and to invoke digital sharpening processes only when needed.
From an extensive study to determine whether it is possible to predict from data collected at the time the original scene is photographed, the probability and severity of the sharpness defect that will be present in the final image, it was discovered that the extent of the sharpness defect depends primarily on the following factors: reproduction magnification, final image viewing distance, camera lens focal length, DSC resolution or camera film speed, shutter time, subject motion, photographer hand tremor, and subject distance if outside of focus range. In the present invention, these factors are used to predict, on an image by image basis, the severity of the sharpness defect, and that information is transferred from the camera to the photofinishing system where it commands the automatic printer control system, or in the case of human assisted printers, alerts the operator, to apply sharpness defect location and correction techniques only when warranted, thereby improving picture quality and enhancing photofinishing productivity.
One proposal for optimizing multiple image processing operations is described in U.S. Pat. No. 5,694,484, issued Dec. 2, 1997, to Cottrell et al. Cottrell et al. disclose an image processing system that proposes to optimize the perceptual quality of images undergoing a series of image-processing operations selected by an operator. The system consists of a set of selected image-processing operations, an architecture, and a control system. These elements take into consideration profiles of source characteristics from which the images are generated, profiles of output device characteristics, and the impact that image processing operations (individually or in concert) will have on perceived image quality. Control parameters for the individual image processing operations are modified by optimizing an image quality metric (a single numerical quality) based on mathematical formulas relating objective metrics (such as sharpness, grain, tone, and color) with perceived image quality. In the method described by Cottrell et al., the values for the individual control parameters are varied over useful ranges until the image quality metric achieves an optimal value. Besides involving significant computation resources to evaluate the multitude of parameter permutations, this method requires operator intervention to select the set of image processing operations that will be applied in each case.
In U.S. Pat. No. 5,835,627, issued Nov. 10, 1998 to Higgins, Hultgren and Cottrell, the process described above in the '484 patent is refined with the addition of an algorithm selector that tries each possible sequence of image processing operations and a customer satisfaction index (CSI), which proposes to balance the perceived image quality and the image processing time, as exhibited by the different image processing sequences. As was the case in U.S. Pat. No. 5,694,484, the image quality estimate is based on device profiles that are constant value inputs for each image source and downstream device, and that are typically generated during calibration of the individual devices in a factory or laboratory setting (see U.S. Pat. No. 5,835,627, col. 3, line 60–65). Besides involving significant computation resources to evaluate the multitude of parameter permutations as in the '484 patent, this refinement increases the amount of computation by causing the process to iterate through each new sequence until an optimal CSI is obtained.
Despite the elaborate methodology disclosed in U.S. Pat. Nos. 5,694,484 and 5,835,627, such systems fail to recognize the importance and use of capture-specific data, that is, variable data collected at the time of image capture, to predict on an image by image basis the best selection of image defect correction algorithms to apply. In particular, it would be desirable to make advantageous use of scene- and exposure-specific data to predict the best selection of image defect correction algorithms to apply.